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(57) Abstract: A telestrator system is disclosed that al- 
lows a broadcaster to annotate video (10) during or after 
an event For example, while televising a sporting event, 
an announcer (or other user) can use the present inven- 
tion to draw over the video (10) of the event to highlight 
one or more actions, features, etc. In one embodiment, 
when the announcer draws over the video ( 1 OX it appears 
that the announcer is drawing on the field or location of 
the event. Such an appearance can be performed by map- 
ping the pixels location from the user's drawing to three 
dimensional locations at the event Other embodiments 
include drawing on the video (10) without obscuring per- 
sons and/or other specified objects, and/or smoothing the 
drawings in real time. 
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TELESTRATOR SYSTEM 

CROSS-REFERENCE TO RELATED APPLICATIONS 
5 This Application is related to the following Applications: 

SYSTEM FOR ENHANCING A VIDEO PRESENTATION OF A LIVE 
EVENT, by Gloudemans, et al., Serial Number 09/160,534, filed September 24, 
1 998, Attorney Docket No. SPTV 1 0 1 7MCF/BBM; 

DETECTING A TALLIED CAMERA, by Marvin S. White, et al., Serial 
10 Number 09/1 60,532, filed September 24, 1998, Attorney Docket No. SPTV1 01 3 
MCF/BBM/WJH; and 

BLENDING A GRAPHIC, by James R. Gloudemans, et al., Serial Number 
09/160,428, filed September 24, 1998,' Attorney Docket No. SPTV1019 
MCF/BBM/WJH. 

1 5 Each of these related Applications are incorporated herein by reference. 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention is directed to a system for annotating video or still 
20 images. 

Description of the Related Art 

The remarkable, often astonishing, physical skills and feats of great 
athletes draw millions of people every day to follow sports. In particular, the 
25 amount of people watching sports on television and the amount of advertising 
revenue received for televised sports has increased significantly. To satisfy the 
increased demand for televised sports, broadcasters have deployed a varied 
repertoire of technologies to highlight to viewers these exciting events. For 
example, broadcasters have started adding graphical enhancements to the video 
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of the sporting events. Examples of graphic enhancements have included 
highlighting moving objects, highlighting portions of a playing field (e.g. first 
down line), adding virtual advertisements and adding of other graphics to the 
video of the event. 

5 One enhancement that has been particularly popular has been the 

telestrator system, which is a system that allows an operator to draw on live or 
stored video. Examples of drawing on the live or stored video include the 
operator drawing lines, circles or other shapes, or symbols. While the operator 
draws the lines, the operator will discuss events, objects or people significant to 

1 0 the drawing. One example of a successful use of telestrator systems is during the 
broadcast of American football games. An announcer may draw on an image to 
show a player out of position, a loose ball, a player offsides, etc. Various 
telestrator systems allow the broadcaster to control the width, color, texture and 
shadowing of any drawing. Additionally, some systems allow for the inclusion 

15 of automatic shapes and animated icons with the video. 

Prior art telestrator systems typically include a video monitor with a touch 
screen or digitizing table. In communication with both the touch screen and the 
monitor is a processor. The processor receives input from the touch screen and 
is responsible for adding to the video whatever is drawn by the user. 

20 While the prior art telestrator systems have been successful, they can be 

improved to further enhance viewer enjoyment of the event. For example, the 
prior art telestrator systems only work well when drawing on still images from a 
video. If the video images move, the camera moves or the broadcaster switches 
to a different camera after the operator draws on the video, then the drawing from 

25 the operator has to be removed from the video or the drawing will be out of place. 
This is because the drawing is added to the video by overlaying it on top of the 
current frame being broadcast. Even if the images or the camera move, the 
position of the drawing in the frame will not move and will remain in the same 
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pbsition in subsequent frames of video. For example, Figure 1 depicts a frame of 
video 10. Frame 10 shows a portion of an American football field including the 
15 yard line and the 20 yard line. In this example, the operator of the prior art 
telestrator system has drawn an arrow 12 to indicate where a player was running, 
5 direction of movement, etc. If the camera providing the video were to pan so that 
the camera points to a different portion of the football field, the portion of the 
football field in the video would change but the position of arrow 12 in the frame 
would not. For example, Figure 2 shows a second frame 14 from the prior art 
telestrator system. Frame 14 differs from frame 10 because the camera has 

10 panned to show the 25 yard line. Although the location of the portions of the 
field have moved between frame 10 and frame 14, the position of arrow 12 
remains in the exact same position in frames 1 0 and 1 4. If the operator originally 
was drawing arrow 12 to show the path a player ran on the field, when frame 14 
is depicted arrow 12 will no longer show the actual path. That is, in frame 10 

15 arrow 12 shows a path across the 15 yard line stopping just before the 20 yard 
line. On the other hand, in frame 14 arrow 12 shows a path not crossing the 15 
yard line, but rather crossing the 20 yard line. Thus, arrow 12 is no longer an 
accurate representation of what the operator intended to draw. 

A further enhancement that could be made to prior art telestrator systems 

20 is to improve the quality of the illustration. For example, an operator may have 
sloppy penmanship and, thus, the lines or shapes drawn could appear too sloppy 
or not pleasing to the viewer. Alternatively, when prior art telestrator operators 
draw over video, whatever they draw tends to be placed on top of and obscuring 
the players or objects. Therefore, there is a need to improve the prior art 

25 telestrator systems. 
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SUMMARY OF THE INVENTION 
The present invention, roughly described, is directed to an improved 
telestrator system that allows a broadcaster or other operator to annotate video 
during or after an event. For example, while televising a sporting event (or other 
5 type of event), an announcer (or other user) can use the present invention to draw 
over the video of the event to highlight one or more actions, features, etc. In one 
embodiment, when the user draws over the video it appears that the user is 
drawing on the actual field or location of the event. Other embodiments of the 
present invention include technology for smoothing out whatever the user draws 
1 0 to make it look more pleasing to the eye. In yet another embodiment, the present 
invention includes drawing in a manner that does not obscure or hide persons or 
objects. 

In one embodiment, the system receives points drawn by a user. These 
points are used to create a smooth curve and the curve is transformed to three 

1 5 dimensional locations on the field or surface. These three dimensional locations 
are then transformed back to two dimensional positions in future video frames so 
that whatever is drawn will appear to be drawn on the field even if the camera 
moves or a different camera is used. 

In one embodiment, the system includes a touch screen and a display in 

20 communication with one or more processors. The processors) receive data 
representing the user's drawing from the touch screen, smooth the data, transform 
the data to three dimensional locations, transform the data back to two 
dimensional locations for future frames or fields, and blend the user's drawing 
with the video. The blending can be performed with a video processor, a keyer 

25 or another video modification device. 

In order to transform the drawings between two dimensional and three 
dimensional coordinates, the processors) use data from camera sensors. These 
camera sensors are connected to or in communication with various video cameras 
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at the event. Suitable camera sensors include optical shaft encoders, 
inclinometers and fibre optic gyros, as well as monitoring electrical outputs of the 
camera lens (2X Extender, focus, zoom). Data from the camera sensors is used 
to determine the camera viewpoint (position in space, orientation and field of 
5 view). This data can be used to create transformation matrices to transform 
positions between three dimensional real space and two dimensional video 
coordinate systems. 

The present invention is accomplished using a combination of both 
hardware and software. The software used for the present invention is stored on 
10 one or more processor readable storage media including hard disk drives, CD- 
ROMs, optical disks, floppy disks, RAM, ROM or other suitable storage devices. 
In alternative embodiments, some or all of the software can be replaced by 
dedicated hardware including custom integrated circuits, gate arrays, FPGAs, 
PLDs, and special purpose computers. Additionally, much of the hardware 
discussed below can be replaced by additional software on general purpose or 
special video computers. 

These and other objects and advantages of the present invention will 
appear more clearly from the following description in which the preferred 
embodiment of the invention has been set forth in conjunction with the drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figures 1 and 2 show video frames that result from the use of prior art 
telestrator systems. 

Figures 3 and 4 depict video frames that result from use of the present 
invention telestrator system. 

Figure 5 is a block diagram of one exemplar set of hardware used to 
implement the present invention. 

Figure 6 is a flow chart describing a process that uses the current 
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Figure 7 is a flow chart describing the setting of a filter pass band. 
Figure 8 is a flow chart describing a portion of the operation of the present 
invention. 

5 Figure 9 is a flow chart describing the process of smoothing a curve. 

Figure 10 is a flow chart that explains one embodiment of the process of 
determining alpha values for pixels in a video. 

Figure 1 1 is a flow chart describing a portion of the operation of the 
present invention. 

10 Figure 12 is a flow chart describing the steps for enhancing a field of 

video. 

DETAILED DESCRIPTION 
The present invention can be used to enhance a video representation of an 

1 5 event. For purposes of explanation only, the example discussed in this present 
application is the broadcast of an American football game. However, .the present 
invention applies equally to other sporting events and other non-sporting events. 
Additionally, the present invention can be used to enhance live video, stored 
video or still images. One exemplar enhancement is the blending of one or more 

20 graphics with the video. A graphic includes drawings or illustrations, highlights, 
logos, shapes, etc. Examples of drawings include curves, shapes, text and 
anything that can be drawn by hand or otherwise. A curve includes curved lines 
and straight lines. Video means the analog or digital signal depicting (or used to 
produce) moving images. Blending means combining at least a first image or 

25 video with at least a second image of video such that the result includes all or part 
of that first image or video and all or part of the second image or video. One 
example of how images are blended includes using a keyer to key one video or 
image over another video or image. 
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One embodiment of the present invention telestrator system allows a user 
of the system to draw on the video such that the drawing appears to be drawn 
directly on the surface of the scene being depicted in the video. For example, 
during an American football game, if a user draws on the video it appears that the 
5 user is actually drawing on the playing field. After the user draws the illustration, 
the drawing will appear to be painted on the playing field so that if the camera 
moves or the broadcaster changes cameras the drawing still appears on the 
playing field in the appropriate perspective. This feature is depicted in Figures 
3 and 4. Figure 3 shows a video frame 1 6 which is broadcast as a result of using 

10 the present invention. The operator had drawn an arrow 12 to show the path of 
a player or object on the playing field. Figure 4 shows a video frame 1 8 which 
is also the result of using the present invention. However, Figure 4 shows arrow 
1 2 at a different position in frame 1 8 than in frame 1 6. That is because after the 
user had drawn arrow 12 and it was depicted in frame 16, the camera panned 

15 down the playing field. Instead of being pointed between the 20 and 15 yard 
lines, the camera is now pointed directly at the 20 yard line and, thus, the 25 yard 
line is now visible. Because the field moved within the camera's field of view, 
the location of arrow 12 also moved within the camera's field of view. Arrow 12 
now appears to be drawn directly on the playing field as opposed to on the 

20 television monitor. As the field moves on the video screen, arrow 12 will also 
move. 

In some embodiments, the blending of the graphics from the user of the 
telestrator system must take into account occlusions. That is, if a player steps on 
top of the area where the graphic is, the graphic should not be drawn over the 
25 player. The player should appear to be stepping or standing in front of the 
graphic. Similarly, there may be portions of the surface or field which should not 
be drawn over by the user's graphic. For example, the broadcaster may decide 
that the announcer's drawing should not appear to obscure any objects (such as 
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a football) or markings on the field (e.g. the numbers painted on the field). 

Fig. 5 is a block diagram of the hardware components that comprise one 
embodiment of the present invention. The user interface for the operator of the 
present invention includes a display 1 02 and touch screen 1 04 mounted on display 
5 1 02. In one embodiment, the display is a flat panel display and the touch screen 
is manufactured from Elo TouchSystems, Inc. (www.elotouch.com) . The output 
of touch screen 104 is sent to telestration processor 108. In one embodiment, 
telestration processor 108 sends a video signal to video format converter 106, 
The output of video format converter 1 06 is communicated to display 1 02. In one 

1 0 embodiment, video format converter 1 06 converts the video signal of telestration 
processor 1 08 from VGA format to NTSC format. In other embodiments, there 
may not be a need for converter 106. In operation, a user of the system can use 
a finger on touch screen 1 04 to draw a graphic (e.g. shape or curve). Information 
about what is being drawn is sent to telestration processor 108. The resulting 

15 blend of the graphic with the video is sent back to display 102. In another 
embodiment, display 102 would receive the actual broadcast. 

Telestration processor 108 also receives the program video. At a given 
event, a broadcaster typically uses multiple cameras. The producer or director of 
the event will typically choose one camera for broadcast. The program video is 

20 the video from the camera chosen for broadcast. Telestration processor 1 08 may 
also receive video from other sources (e.g. replay decks, other cameras, etc.) for 
offline work. Telestration processor 108 also receives data from gatherer 110. 
In one embodiment, gatherer 1 1 0 is a computer. In another embodiment, gatherer 
1 10 can be dedicated hardware. Gatherer 110 receives time code from VITC 

25 (Vertical Interval Time Code) 112, camera sensor data from the various 
instrumented cameras (line 1 14), and key data from a key data processor 1 16. 

Each of the cameras used with the present invention includes a number of 
camera sensors that measure data (camera sensor data) to determine the field of 
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view of the camera. Fig. 5 shows an example of camera 130 with camera sensors 
1 32. The camera sensors could include any or all of the following: optical shaft 
encoders, fibre optic gyros, inclinometers, and reading voltages from the lens (e.g. 
2X Extender, focus, zoom). Other camera sensors can also be used. Data from 
5 camera sensors 132 is sent to gatherer 110. In one embodiment, the camera 
sensor data for a given camera is transmitted to a production studio from the 
camera location via the camera's audio channel. The production studio includes 
hardware to demodulate the audio channel and send the camera sensor data to 
gatherer 1 10. In some instances, the production studio is in a truck at the event. 

10 The video from camera 130 is sent to camera control unit 134, which 

controls various video and optical parameters for camera 130. The output of 
camera control unit 134 is sent to VITC inserter 136 which adds a time code and 
unique camera identifier into the vertical blanking interval of the video from 
camera 130. The present invention can be operated using one or more 

1 5 instrumented cameras. In one embodiment, the present invention is operated with 
six cameras instrumented as shown in Fig. 1. Each camera would have its own 
CCU and its own VITC inserter. Each camera's VITC inserter is synchronized 
with VITC 112. In alternative embodiments, the present invention can be used 
with fixed, non-instrumented cameras. In another alternative, the present 

20 invention can be used with non-instrumented cameras that are not fixed, in 
combination with image recognition. 

Broadcast cameras used according to the present invention are typically 
part of a camera assembly that includes a tripod base, a camera head interface 
(also called a tripod head interface) mounted on the tripod base, a camera head 

25 (also called a tripod head) mounted on the camera head interface, a camera 
mounted on the camera head, and a lens mounted on the camera. The tripod base 
is the standard tripod known in the art. The camera head can be a standard 
camera head known in the art, for example, a Vector 70 from Vinten, Inc. The 
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Vector 70 allows the camera to tilt and pan. To measure tilt and pan, optical shaft 
encoders can be attached at the pan axis and tilt axis to measure the rotation of the 
camera about the pan axis and tilt axis. An example of an optical shaft encoder 
is the CP-850 Series from Computer Optical Products, 9305 Eton Avenue, 
5 Chatsworth, California 91311. The pan shaft encoder is mounted in the camera 
head interface. 

One or more inclinometers can be mounted on the camera head interface 
to measure attitude of the stationary portion of the camera assembly. Thus, if the 
camera assembly is on an angle, is kicked, or otherwise changes, that change in 

10 attitude will be detected. One example of a suitable inclinometer uses liquid 
between a pair of plates, and measures change of capacitance. Another example 
is an electrolyte varying the conductance between two conductors. In one 
embodiment, a suitable inclinometer indicates an absolute angle. In one 
alternative, the inclinometer can indicate angles up to ± 1 °,± 1 .5 ° or ± 6 ° . Other 

15 suitable ranges can also be used. An example of a suitable inclinometer is the 
Ceramic Tilt Sensor SH50054 from Spectron, 595 Old Willets Path, Hauppaug, 
New York 1 1 788, (5 1 6) 582-5600. Other suitable inclinometers can also be used 
with the present invention. In one embodiment, the system will include two 
inclinometers. A first inclinometer would be mounted on a first surface of the 

20 camera head interface. A second inclinometer would be mounted on a second 
surface of the camera head interface. The first surface is ideally orthogonal to the 
second surface. Both inclinometers would be connected to an electronics board 
for receiving the data and packaging it with the other sensors. The electronics 
board is part of the camera head interface. 

25 One or more gyros can be mounted on the camera lens to measure high 

frequency vibrations and mechanical compliance between the moving portion of 
the camera assembly and the stationary portion of the camera assembly. In one 
embodiment, the system includes two gyros mounted on a block. The block itself 
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is mounted on the lens of the camera. The first gyro is mounted on a first surface 
of the block and the second gyro is mounted on the second surface of the block. 
The first surface is ideally orthogonal to the second surface. In an alternative 
embodiment, the two gyros can be mounted on the tripod or the camera head 
5 interface. While the encoders discussed above measure angles relative to a fixed 
base, the gyro measures absolute angular rate information that is not relative to 
any part of the camera assembly. In one embodiment, the gyros are fibre optic 
gyros. An example of a suitable fibre optic gyro is the E-Core 2000 Series Fibre 
Optic Gyro manufactured and sold by KVH Industries, Inc., 50 Enterprise Center, 

10 Middleton Road, Rhode Island 02842. Other gyros that could also be used 
include a ring laser, mechanical gyro, tuning fork, spinning disk gyro, 
semiconductor gyro, etc. Integrating the output of the gyro will produce an angle 
delta indicating that the gyro was rotated by a certain angle. The fibre optic gyro 
discussed above measures angular rate about a single axis. However, multi- 

1 5 directional gyros can also be used. In one embodiment, the system can use three 
or more gyros instead of two in order to measure pitch, roll and yaw. 

In one embodiment, each camera sensor unit 132 includes electronics in 
the camera head interface for receiving signals from the pan encoder, tilt encoder, 
two gyros, two inclinometers, 2X Extender, lens focus and lens zoom. This data 

20 is sent to a processor which packages the data, synchronizes the data and encodes 
the data on the audio channel by modulating an audio carrier in order to send the 
data in the camera's microphone (or audio) channel to the television production 
studio. More information about using camera sensors can be found in United 
States Patent Application No. 09/1 60,534 filed September 24, 1 998, United States 

25 Patent No. 5,9 1 2,700, both of which are incorporated herein by reference. 

Gatherer 1 1 0 also receives information from key data processor 116. In 
one embodiment, key data processor 116, telestration processor 1 08 and rendering 
processor 1 50 are 02 workstations from Silicon Graphics. Other computers can 
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also be used. Key data processor 1 16 is used to provide information on which 
colors are included and excluded from being blended with the illustrations by the 
operator. More information on this will be discussed below. In sum, gatherer 1 1 0 
packages the key data, the sensor data and the time code into a set of information 
5 and sends that set of information to telestration processor 1 08. Gatherer 11 0 can 
package other types of data as well. Telestration processor 108 determines how 
the graphics drawn on touch screen 104 should be blended with the program 
video and sends that information to rendering processor 150. Rendering 
processor 150 takes charge of the actual blending of the graphics with the 

10 program video. 

To perform the blending, rendering processor 1 50 supplies two signals 1 70 
and 1 72 to program keyer 1 52 and off-line keyer 1 54. Signal 1 70 is called alpha 
and signal 172 is called foreground. Program keyer also receives an input from 
video delay 158. Video delay 158 receives the program video and delays it for 

15 a number of frames and transmits the delayed video to program keyer 152 and 
rendering processor 1 50. The delayed program video sent to program keyer 1 52 
is called background. Based on the level of the alpha 170 from rendering 
processor 150, program keyer 152 determines how much foreground and 
background to blend on a pixel by pixel basis. Program keyer 1 52 can blend from 

20 1 00% foreground and 0% background to 0% foreground and 1 00% background. 
In one embodiment, the alpha for a particular pixel can range from 0% to 1 00% 
(or 0- 1 or another similar range as per the specification of the keyer). The output 
of program keyer 1 52 can be sent for broadcast, recording or both. During typical 
operation described below, the output of program keyer 1 52 is live video. 

25 Alpha 170 and foreground 172 are also sent to off-line keyer 154. The 

background signal received by off-line keyer 1 54 is from video delay 1 60. In one 
embodiment, the delayed video from video delay 160 is from a video storage 
device such as a tape player or other suitable storage device. In another 
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embodiment, the video input to delay 160 is from a camera. In one alternative, 
the video input to delay 160 is also sent to telestration processor 108. The output 
of offline keyer can be used to produce replays, half time shows or highlights. 
The system can also include a kill switch which can be used by an operator 
5 to enable or disable the keying of the graphic. Additionally, a watch dog timer 
can be included to automatically disable the keying of the graphic if rendering 
processor 150 stops sending a periodic signal. 

As an additional option, the system could also include a data inserter for 
inserting non- video data into a television signal. Non-video data is information 

1 0 other than traditional data used by a television to draw the normal scan lines on 
a television display. An example of non-yideo data is data transmitted during the 
vertical blanking interval, which can be closed-caption data, statistics regarding 
the game, interactive queries or Internet addresses. The optional data inserter can 
receive the video signal from program keyer 1 52 and insert the non- video data 

1 5 into the vertical blanking interval of the television signal. The output of the data 
inserter would be broadcast, recorded or both. In one embodiment, "a data inserter 
can insert into the video signal instructions for a computer to enhance the video. 
At the viewer's home will be a set-top box which can read the instructions from 
the received signal and pass the information to the computer. The computer can 

20 use the instructions to blend the graphic with the video. Thus, a viewer can 
customize and control the enhancements using the viewer's personal computer. 
Alternatively, the set-top box will be capable of applying the enhancement 

Figure 6 is a flow chart describing operation using the present invention. 
One embodiment of the present invention includes making the graphics drawn 

25 using touch screen 104 look like they were drawn on the actual field or surface. 
This requires transforming coordinates between two dimensional coordinate 
systems and three dimensional coordinate systems. One means for accomplishing 
such transformation includes using a model of the surface being displayed on 
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television. In step 250, the user of the system creates a mathematical model of the 
environment being displayed on the television. If the system is being used to add 
illustrations to a football field, then the environment would only include the 
football field and step 250 would include creating a model of the football field. 
5 If the user intends to add illustrations to other portions of the stadium, then the 
environment must include those other portions of the stadium as well. The model 
created is a three dimensional model of the environment. For example, if the 
environment to be modeled is a football field, the model would include a 
description of the surface of the football field. Most football fields are not flat 

10 surfaces, and include crown for drainage purposes. Additionally, many fields 
include other variations in the height (and possibly length and width) of the field 
due to errors and other abnormalities. Thus, the model will serve as a three 
dimensional representation of the surface of the field. If the environment includes 
portions of the stadium, then the model will include the relevant features of the 

1 5 stadium such as any retainer walls, the top of the stands and any other surface the 
user may want to add a graphic to. 

One means for determining a model is to measure data for different points 
in the environment. In one embodiment, each data point includes x, y and z 
values. Any method can be used to obtain these x, y and z values. One example 

20 of a suitable method is to use a laser plane for z values and a laser range finder for 
x and y values, or other surveying devices. For simplicity, in a broadcast of an 
American football game, a coordinate system is set up with the origin at the near 
corner of the left end zone, the y-axis along the width of the field (e.g. the back 
of the end zone), the x-axis is along the length of the field (e.g. the side line) and 

25 the z-axis extends vertically from the field. The operator can measure or use the 
yard markings on the field to determine the x and y coordinates for most points 
of interest on the field. The laser plane can be used to measure the corresponding 
z coordinate. The laser plane is utilized by placing the laser plane at the origin (or 
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another point) and reading the laser image off a pole that is positioned at the point 
of interest In one embodiment, data samples are taken for the back of both ends 
zones, both goal lines, both 20 yard lines and both 40 yard lines. For each yard 
line measured, measurement should at least be taken at each side line and then one 
5 or more points between the sidelines, including the middle of the field. 
Additional data points around the yard lines can also be measured. If the 
environment includes parts of the stadium, the laser plane, measuring tape or 
another measuring device can be used (as well as geometry) to determine data for 
other points in the environment. 

10 In one embodiment, the data points measured can be used to create the 

model. That is, data points can be plotted and connected (symbolically). In 
another embodiment, a set of curves are created using the measured data. That 
is, if data is taken for a number of points along a line, then that line can be 
modeled by fitting the data into a curve of the form A + By + Cy 2 = z, or using a 

1 5 different suitable equation. Thus, any point on a line at a given yard position can 
be found using that equation because it is assumed that every point at the yard 
position has the same x value. As the y value changes, the z value will also 
change. Similar curves can also be used to represent other lines. If the system 
wants to find the z value for a point between two curves, the system can 

20 interpolate. The set of curves created constitutes the model. Some models may 
require information in addition to the curves (e.g. polygons and/or topology). 

After creating the model in step 250, the system is registered in step 252. 
Registration, a technology known by those skilled in the art, is the process of 
defining how to interpret data from a sensor and/or to ascertain data variable for 

25 operation of the system. The sensors described above output data, for example, 
related to position and orientation. Since position and orientation are relative, the 
system needs a reference from which to determine position or orientation. Thus, 
in order to be able to use camera sensor data, the system needs to know how to 
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interpret the data to make use of the information. Generally, registration includes 
pointing the instrumented cameras at known locations and solving for unknown 
variables used in transformation matrices. More detail of how to register the 
system can be found in any of the patent applications or patents incorporated by 

5 reference above or in United States Patent No. 5,862,5 1 7 which is incorporated 
herein by reference. 

In step 254, the operator will set up inclusions and exclusions. In one 
embodiment of the present invention, the graphic can be added to the video 
without taking into account the contents of the video signal. There will be no 

1 0 accounting for occlusions; for example, a player or object in front of the surface 
on which the enhancement is intended to appear. In another embodiment, the 
present invention can include inclusions and/or exclusions in order to account for 
occlusions and other object or markings. An inclusion is a color range for a pixel 
that can be enhanced using the present invention. An exclusion is a color range 

15 for a pixel that should not be enhanced using the present invention. During 
operation, the operator can set up one or more inclusions and/or one or more 
exclusions. For example, the operator may decide that a graphic can be drawn 
over green (grass) and brown (dirt). Additionally, the operator may want to set 
up the exclusion so that a line is not drawn over a specific color (e.g. team's 

20 uniforms). It is possible to allow drawing over one shade of green (grass) and not 
allow drawing over a second shade of green (team's shirt color). In an alternate 
embodiment of the present invention, exclusions can also include video frame 
pixel positions or real world locations that are not to be enhanced. The process 
of setting up inclusions and exclusions is performed using key data processor 1 1 6. 

25 When operating the system of Figure 5, step 254 of Figure 6 includes 

sending an output from a camera to key data processor 1 16. The camera will be 
panned and tilted to point to the different areas of the field or stadium. The 
operator can view the output of the camera on a monitor and, using a pointing 
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device (e.g. a mouse), select areas for inclusion (create an inclusion filter) or 
exclusion (create an exclusion filter). For example, the operator could choose the 
shady grass, sunny grass and dirt for inclusions. The operator may choose the 
players* uniforms, shoes, football and chalk as exclusions. 
5 When setting up inclusions and exclusions, key data processor 1 1 6 first 

receives a set of pixels. In one embodiment of the present invention, the pixel set 
received is from the output of one of the cameras. For example, a pixel set can 
include selected pixels from an image of the playing field, selected pixels from 
an image of one of the teams * uniforms, or other images. In another embodiment, 

10 the pixel set can be received from a stored image. 

Once the pixel set is received, the operator of key data processor 116 
determines whether the pixel set is to be used for identifying exclusion pixels or 
identifying inclusion pixels. If it is determined that the pixel set has been received 
for establishing criteria for exclusion pixels, then key data processor 116 

15 generates an exclusion filter. If it is determined that the pixel set has been 
received to establish criteria for inclusion pixels, then key data processor 1 16 
generates an inclusion filter. Multiple inclusion filters and exclusion filters can 
be generated. 

In one embodiment of the present invention, key data processor 116 
20 generates inclusion filters and exclusion filters by generating a set of histograms 
characterizing the received sets of pixels. For example, an inclusion filter may 
include a Y histogram, a Cr histogram and a Cb histogram, all of which describe 
the inclusion filter in YCbCr format. The Y characteristic histogram has a 
horizontal axis representing luminance values and a vertical axis representing the 
25 number of pixels in the received pixel set that corresponds to each of the 
luminance values. The Cr characteristic histogram has a horizontal axis 
representing Cr values and a vertical axis representing the number of pixels in the 
received pixel set that corresponds to each of the Cr values. The Cb characteristic 
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histogram has a horizontal axis representing Cb values and a vertical axis 
representing the number of pixels in the received pixel set that corresponds to 
each of the Cb values. Each histogram has a respective pass band that defines the 
Y, Cr, or Cb characteristics that a pixel must have to be an inclusion pixel. 
5 Accordingly, a pixel will be designated as an inclusion pixel when the filter is 
applied and the pixel has a Y characteristic value within the Y pass band, a Cr 
characteristic value within the Cr pass band, and a Cb characteristic value within 
the Cb pass band. Exclusion filters work in a similar manner. 

Figure 7 illustrates a sequence of operation performed by key data 

10 processor 1 16 to determine a pass band for an inclusion filter histogram or an 
exclusion filter histogram. In the embodiment using YCbCr, the steps of Figure 
7 are performed for each of the three histograms. First, key data processor 116 
identifies the most frequently occurring value for the characteristic (Y, Cr, or Cb) 
represented by the histogram in step 270. Next, the characteristic value is 

15 incremented in step 272. It is then determined whether the number of pixels 
having the resulting characteristic value is within a predetermined percentage of 
the number of pixels having the most frequently occurring characteristic value in 
step 274. In one embodiment of the present invention, the predetermined 
percentage employed in step 274 is 10 percent for an inclusion filter and 50 

20 percent for an exclusion filter. 

. If it is determined that the number of pixels with the characteristic value 

is above the predetermined percentage, then the characteristic value is 

♦ 

incremented in step 272 and a new comparison is performed. If it is determined 
that the number of pixels with the characteristic value is not above the 
25 predetermined percentage, then the maximum characteristic value for the pass 
band is set in step 276 to be equal to the last characteristic value with a number 
of pixels above the predetermined percentage. 

Once the maximum pass band characteristic value is set, the characteristic 
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value is set to be equal to the characteristic value just below the most frequently 
occurring characteristic value in step 278. It is then determined whether the 
number of pixels having the resulting characteristic value is above a 
predetermined percentage of the number of pixels having the most frequently 
5 occurring characteristic value in step 280. In one embodiment of the present 
invention, the predetermined percentage employed in step 280 is one percent for 
an inclusion filter and twenty five percent for an exclusion filter. In another 
embodiment of the present invention, the predetermined percentage employed in 
step 280 is 1 0 percent for an inclusion filter and 50 percent for an exclusion filter. 

10 If it is determined that the number of pixels with the characteristic value 

is above the predetermined percentage, then the characteristic value is decreased 
in step 280 and a new comparison is performed. If it is determined that the 
number of pixels with the characteristic value is not above the predetermined 
percentage, then the minimum characteristic value for the pass band is set in step 

15 284 to equal the last characteristic value with a number of pixels above the 
predetermined percentage of the number of pixels representing the most 
frequently occurring characteristic value. 

Although the generation of an inclusion filter and exclusion filter has been 
described with respect to forming a histogram, one of ordinary skill in the art will 

20 recognize that it is not necessary to actually form a graphical image of a 
histogram. A processor could also maintain a table of data that reflects the Y, Cr, 
and Cb data. It will also be recognized that percentage thresholds identified 
above are not the only percentages that may be employed. Any number of 
percentages may be employed, depending upon the resolution that is desirable for 

25 the filter. One with ordinary skill in the art will further recognize that other 
methods can be employed for generating inclusion filters and exclusion filters. 
For example, a color region or set of color regions can be selected for inclusion 
or exclusion using a chromacity diagram. More information on how to set up and 
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use inclusions and exclusions is provided in United States Patent Application No. 
09/160,534, filed September 24, 1998, incorporated herein by reference. 

After the inclusion/exclusion filters are created, a color map is created. 
The color map is a database with a record for each possible color based on Y, Cr 
and Cb values. In an alternative embodiment, less than all of the possible colors 
are represented in the color map. For each color in the color map, the database 
stores an alpha percentage. The stored alpha percentage could be a number 
between zero and one hundred, zero and one, or another suitable range. Although 
the term percentage is used, the stored number need not be a number designating 
a fraction of one or a fraction of one hundred. The stored number need only 
designate an amount ofblending. In one embodiment the stored alpha percentage 
is a number between 0 and 255. When the operator selects a pixel set for an 
inclusion or an exclusion, the operator is asked to provide a percentage between 
zero and one hundred percent. The number provided by the operator is converted 
to a scaleof 0-255. The converted number is stored as the alpha percentage for 
each color described by the inclusion/exclusion filter (e.g. each color within the 
pass bands of the filter). 

Additionally, a taper zone is set up. The taper zone includes colors outside 
of but near the pass bands for a filter. One method for determining the colors in 
the taper zone is to create an additional set of pass bands (in addition to the 
inclusion pass bands) in the same manner as described above, but use different 
percentages so that the pass bands are wider. Those colors in the new pass bands 
but not in the inclusion pass bands are in the taper zone. The colors in the taper 
zone closest to the inclusion pass bands are given an alpha percentage equal to or 
slightly lower than the alpha percentage given to the colors in the 
inclusion/exclusion pass bands. The colors in the taper zone farthest from the 
inclusion pass bands are given an alpha percentage equal to no blending. The 
colors in between are given an alpha percentage based on linear interpolation. An 
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analogous method is used for taper zones near exclusions. The alpha percentages 
for each color in the taper zones are stored in the color map. 

After step 254 of Figure 6, the present invention is operated in step 256 
according to the methods described in Figures 8-12. Figure 6 also shows step 258 
5 which includes updating the exclusions and inclusions using key data processor 
1 1 6. During an event, lighting conditions, field conditions and weather conditions 
may change causing the colors to be included and excluded to change. In that 
case, the inclusions and exclusions could be updated in step 258. Note that Figure 
6 shows step 258 being performed after step 256. However, steps 256 and 258 are 

10 actually performed concurrently or otherwise overlapped in time. In one 
embodiment or appropriate situations where there are no changes in conditions, 
step 258 would be optional. 

Figure 8 describes the operations performed by telestration processor 1 08. 
Step 300 includes receiving and storing pixel coordinates from touch screen 1 04. 

1 5 That is, as the user touches touch screen 1 04 with the user's finger (or a light pen 
or other suitable device), touch screen 104 outputs the coordinates of the pixel 
touched and sends the coordinates to telestration processor 108. As the user 
continues to move the user's finger, more points are sent to telestration processor 
108. As telestration processor 108 receives those points, they are stored in 

20 memory, a hard disk or another storage medium. Step 300 is not depicted as 
being connected to any other steps because it is a process that is constantly being 
performed during the operation of the system. 

Based on the received program video, telestration processor 1 08 waits for 
a new frame to start in step 302. When a frame starts, in step 304, telestration 

25 processor 108 determines whether it has received any new points from touch 
screen 104 since the last frame. If it has received new points, then in step 308, 
telestration processor 108 smooths the curve represented by the points received 
from touch screen 104. That is, the operator using a light pen, mouse, finger or 
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other pointing device to draw may draw a line or curve that is filled with jagged 
edges otherwise not very pleasing to the eye. Step 308 includes smoothing that 
curve or line to a more pleasing shape. In step 310, telestration processor 108 
creates a set of points to represent the smoothed curve. More details of how a 
curve is smoothed is shown in the flow chart of Figure 9. The steps of Figure 9 
include setting a number of control points. Step 310 includes dividing up the 
curve between each control point into eight (or more) segments. The end points 
of each segment are the points created in step 310. In step 312, telestration 
processor 1 08 identifies the correct set of sensor data. Remember that gatherer 
1 10 receives data from all of the camera sensors and package it into one set of 
data, to be sent to telestration processor 108. Also remember that each VITC 
inserter (e.g. VITC inserter 136) adds a unique identifier to the vertical blanking 
interval of each camera. Thus, telestration processor 108 looks at the vertical 
blanking interval of the program video it is receiving and identifies the camera by 
the unique camera identifier. Gatherer 1 1 0 added a camera identifier to the sensor 
data for each of the individual cameras. Telestration processor 1 08 will match the 
camera identifier and time code from the program video to the camera identifier 
and time code of the appropriate camera sensor data. Thus, telestration processor 
1 08 will use the camera sensor data from the camera providing the program video. 

In one embodiment, the sensor data used to transform the graphics may be 
recently recorded or may be previously recorded (e.g. fifteen minutes previously, 
hours previously, etc.). For example, video captured earlier is stored with its 
camera identifier and times codes. The stored camera identifier and times codes 
can be used to identify the appropriate stored camera sensor data. In this way, the 
invention can operate on stored video and stored sensor data after the event has 
taken place. 

In step 3 1 4, each of the points identified in step 3 1 0 are transformed from 
the two dimensional coordinate system of the touch screen to the three 
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dimensional coordinate system of the football field (or other surface of the live 
event). Various means for converting points between coordinate systems are 
known in the art. One example of doing such a transformation is to use 
transformation matrices. The use of transformation matrices to convert between 
5 coordinate systems is well documented in the art. One of ordinary skill in the art 
would know to insert the data from the camera sensors identified in step 312 into 
a set of transformation matrices and use those matrices. Examples of the use of 
transformation matrices can be found in U.S. Patent No. 5,862,517, U.S. Patent 
No. 5,912,700 and U.S. Patent Application No. 09/160,534, filed September 24, 

10 1998, all of which are incorporated herein by reference. Step 314 includes 
transforming all of the points created in step 310. After step 314, telestration 
processor 1 08 has a set of points in the three dimensional coordinate system of the 
football field (or surface) representing the curve drawn by the operator. In step 
316, the curve is thickened. That is, telestration processor 108 creates a set of 

15 quadrilaterals in three dimensional space. The centers of each of the adjacent 
sides of the quadrilaterals are the points that were transformed in step 314. Step 
316 includes storing the vertices for each of these quadrilaterals. This set of 
vertices defines the thickened line in three dimensional space. In step 318, 
telestration processor 1 08 transforms all the curves to the current video frame (or 

20 field). That is, all of the vertices of the current curve just operated on in steps 
308-3 1 6 are transformed to the two dimensional coordinate system for the current 
camera tallied for broadcast. Additionally, there may be other curves or graphics 
that may be drawn on the screen previously that have been stored in three 
dimensional space. These curves are also transformed in step 3 1 8. The curves are 

25 transformed by transforming each of the vertices of the various quadrilaterals 
representing the curves. The quadrilateral vertices are transformed using 
transformation matrices. 

After transforming all of the curves to the current video frame, the system 
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tessellates the curves in step 320. Tessellation involves breaking up the 
quadrilaterals if the cameras have zoomed in past a certain threshold. One 
example of a threshold is testing whether the sides of the quadrilaterals are more 
than 1 5 pixels. For a particular quadrilateral, if the threshold has been met then 
5 the quadrilateral is subdivided into small quadrilaterals. In one embodiment, the 
system will divide the length and width (in pixels) of the quadrilaterals by 15, 
with the result being the number of sections the length or width must be broken 
up. In one alternative, the number of sections is rounded to the nearest power of 
two for convenience. 

10 In step 322, telestration processor 108 determines alphas for a subset of 

the pixels for the current video frame or field to be blended. These alphas will be 
used to derive the alpha signal 170 sent by rendering processor 150 to program 
keyer 152 and/or offline keyer 154. In one embodiment, step 322 determines 
alpha values for each vertex of each quadrilateral after tessellation (step 320). 

1 5 After step 322, telestration processor 1 08 adds the graphics to the program video 
and sends the enhanced program video to display 1 02. This output sent to display 
102 may not suitable for broadcast. The final broadcast video is provided by 
program keyer 152. After determining the alphas in step 322, telestration 
processor 108 sends the information to rendering processor 150 in step 324 and 

20 the system loops back to step 302. 

If in step 304 telestration processor 1 08 determines that it has not received 
any new points, then an inquiry is made (step 340) as to whether the user has 
finished drawing the curve. In one embodiment, when the user lifts the user's 
finger or light pen from touch screen 104, a pen-up event is generated which 

25 signals that the user has finished drawing the curve or graphic. When there is a 
pen-up event, the method loops to step 344 and the system will save the curve in 
three dimensional coordinates. That is, the points that were the result of step 3 1 6 
are saved. If there was no pen-up event, the method loops to step 348. If 
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telestration processor 1 08 is performing step 348, it is assumed that no new curve 
data has been received and the system needs to redraw the existing curves onto 
the new frame (or field of video). In step 348, telestration processor 108 
identifies the correct sensor data (similar to step 312). After accessing the correct 
5 data in step 348, telestration processor 108 loops to step 318 and continues the 
process from step 3 1 8 to draw the existing curves for the current field or frame 
of video. It is contemplated that the interface for the present invention could 
include an erase button which causes all curves currently being displayed to be 
erased and the system to be reset in that respect. 

10 Figure 9 is a flow chart describing more detail of step 308 of Figure 8, 

smoothing the curve of points received from touch screen 104. In step 402, the 
process receives the pixel coordinates. These pixel coordinates have an X 
coordinate and a Y coordinate pertaining to the position on touch screen 104. In 
step 404, the coordinates are added to a raw point list. The system also maintains 

15 a control point list In step 406, the system determines whether there are any 
points on the control point list. If there are no points on the control point list, then 
the raw points just received in step 402 are added to the control point list in step 
404 and the method loops back to step 402. 

If in step 406, it is determined that the control point list does include one 

20 or more control points, then the method loops to step 410 and telestration 
processor 1 08 determines whether the most recent point added to the raw point list 
in step 404 is a distance greater than a threshold from the last control point added 
to the control point list An example of a suitable threshold is ten pixels. If the 
distance between the most recent raw point and the last current point is not greater 

25 than the threshold, then the method loops back to step 402. If the distance 
between the most recent raw point and the last control point is greater than the 
threshold, then the method loops to step 414 and the most recent raw point is 
added to the control point list If the number of control points in the control point 
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list is exactly two points, then the method loops to step 422 and a line is drawn 
between these two points. After step 422, the method loops back to step 402. If 
in step 418, there are more than two control points in the control point list, the 
method loops to step 426, at which point the next to last control point is removed 
5 from the control point list. In step 430, telestration processor 1 08 will fit a cubic 
Bezier spline through the control points. In step 434, telestration processor 1 08 
will find the removed control point that is farthest away from the curve. A 
removed control point is a point that was once on the control point list, but was 
subsequently removed from the control point list. In step 438, telestration 

10 processor 108 will determine whether distance from the curve to the removed 
control point found in step 434 is greater than a threshold (e.g. ten pixels). If the 
distance is not greater then the threshold, the method loops back to step 402. If 
the distance is less than or equal to the threshold, then in step 440 that removed 
control point is added back to the control point list and the cubic Bezier spline is 

15 re-computed. 

Figure 10 illustrates a sequence of operations performed in one 
embodiment of the present invention for determining an alpha value for one of the 
vertices of a quadrilateral. First, a set of pixels in the program video is selected 
in step 560. In one embodiment, the set of pixels in the program video include the 

20 pixel in the program video having the same coordinates as the vertex under 
consideration. Additionally, the set contains at least four pixels surrounding the 
pixel in the program video having the same coordinates as the vertex under 
consideration. In other embodiments, more or fewer than four additional pixels 
can be used. In one alternative, the four pixels are adjacent to the pixel in the 

25 program video having the same coordinates as the vertex under consideration. In 
other embodiments, the four pixels are a predefined distance away from the pixel 
in the program video having the same coordinates as the vertex under 
consideration. The predefined distance can be selected from experimentation 
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based on the desired effect. In step 562, one of the pixels in the set is selected. 
In step 564, the color of the pixel is used to access the color map. That is the Y, 
Cr and Cb values or characteristics for the pixel are used to access the appropriate 
alpha percentage in the color map described above. In step 566, the alpha 
5 percentage accessed in the color map is added to a counter. 

In step 570, a determination is made as to whether there are more pixels 
to be considered from the set of pixels. If there are more pixels to be considered, 
then a new pixel from the set is selected in step 562. Otherwise, a key fraction is 
calculated in step 572. In one embodiment of the present invention, the key 

1 0 fraction is calculated by dividing the counter value by the total number of pixels 
in the selected set of program pixels. At step 572, the counter value should equal 
the sum of the alpha percentages for each pixel in the set. The key fraction, 
therefore, represents the average alpha percentage for the set. It is contemplated 
that other means can be employed to determine the key fraction. 

15 Once the key fraction is calculated, the alpha value for the vertex is 

determined in step 574 by multiplying the key fraction by a nominal alpha. The 
nominal alpha is set in advance by the user to set the overall transparency or look 
^ of the graphic. In one embodiment, a slider on a graphical user interface allows 
a user to see how changing the nominal alpha changes the keying. The above 

20 described process is repeated for each of the vertices. An alternative method for 
determining alphas and drawing graphic is described in United States Patent 
Application "System For Enhancing a Video Presentation of a Live Event," 
Gloudemans, et al., Serial Number 09/160,534, filed September 24, 1998, 
incorporated herein by reference. 

25 Figure 1 1 is a flow chart which describes the operation of rendering 

processor 1 50. In step 602, rendering processor 1 50 waits for the next frame to 
start. Rendering processor 1 50 receives the program video from video delay 1 58. 
By looking at the program video, rendering processor 1 50 determines when the 
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next frame is starting; Rendering processor 150 receives data from telestration 
processor 108. This data includes the pixel coordinates of the polygons to be 
drawn, tessellations per quadrilateral (number of subdivides), alphas for the 
vertices of the quadrilaterals, the colors of the polygon, texture, fill characteristics 
5 for the polygons and time code for the frame or field associated with the data. 
When rendering processor 150 identifies that the next frame is, it performs the 
step of enhancing the first field (step 604). After enhancing the first field, 
rendering processor then enhances the second field in step 606. After enhancing 
the second field, rendering processor waits for the next frame in step 602. 

10 Figure 12 describes the process for enhancing a field of video. In step 

620, rendering processor 150 determines whether it has data for the field that is 
to be enhanced. That is, by looking at the time code associated with the field from 
the delayed program video, rendering processor 150 looks for the data from 
telestration processor 108 having a similar time code. If that data is found, then 

15 it has data for the new field. Otherwise, there is no data for the particular field 
under consideration. If there is data for the field under consideration, then in step 
622 rendering processor 150 will tessellate the quadrilaterals. Telestration 
processor 108 performed tessellate step 320 previously in Figure 8. When 
telestration processor 108 sent the data to rendering processor 150, telestration 

20 processor 108 sent the alphas for all the vertices; however, it did not send the 
coordinates for the new vertices created during tessellate step 320. Thus, in step 
622, rendering processor 1 50 will tessellate the quadrilaterals and determine the 
appropriate coordinates of the new vertices. In step 624, a flicker filter will be 
applied to the data. 

25 The flicker filter of step 624 is employed to reduce flickering in the 

appearance of the graphic. The flicker filter averages a newly calculated alpha for 
the pixel with past and future alphas for the same pixel to generate a filtered 
alpha. In one embodiment of the present invention, the flicker filter operation is 
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performed according to the following equation: 

F N 



wherein 

a F is the filtered alpha; 
5 X q fp .is a summation of filtered alphas for the selected pixel in prior 

video frames; 

a is the unfiltered alpha of the selected pixel for the current video 

frame; 

Yshj is a summation of unfiltered alphas for the selected pixel for future 
10 video frames; and 

N is a number of values being averaged. 

It is possible to obtain filtered alphas for pixels using future alpha values, 
because delays in the system provide for the calculation of alphas several frames 

15 in advance of their use. In one embodiment, Y, a F? is the sum of the selected 
pixel's filtered alphas for the two video frames preceding the current video frame; 
Ypv is the sum of the selected pixel's filtered alphas for the two video frames 
following the present video frames; and N is equal to 5. In an alternate 
embodiment, the summation of filtered alphas for the selected pixel in prior video 

20 frames (£app) is replaced by a summation of unfiltered alphas for the selected 
pixel in prior video frames. In one embodiment of the present invention, the 
flicker filter operation is applied to the alphas of the center points, border points 
and edge points. In alternate embodiments, the flicker filter is applied to only a 
subset of the alphas of the center points, border points, and edge points. In yet 

25 another embodiment, the flicker filter is applied to each pixel to be blended. In 
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further embodiments, the flicker filter operation is not employed. In still farther 
embodiments, values of alphas from different times can be weighted differently. 

Looking back at Figure 12, in step 626, rendering processor 150 
determines the alpha signal to be sent to the keyers. This step includes taking all 
5 the alpha values and creating a signal in the appropriate video format accepted by 
the keyer. In step 628, rendering processor 1 50 renders the foreground. This step 
includes creating a signal in the appropriate video format accepted by the keyer. 
In one embodiment, steps 626 and 628 occur at the same or overlapping times. 
One embodiment for rendering the foreground and alpha signals includes dividing 

10 the graphic into regions with each region being defined by a set of vertices. In 
one embodiment, the vertices of steps 626 and 628 are the vertices of the 
quadrilaterals determined from the tessellate step. If there was no tessellation, 
then the vertices are the vertices of steps 626 and 628 are the vertices of the 
quadrilaterals without tessellation. 

1 5 For each region, the fill characteristics and alpha of each of vertex of the 

region is employed to establish the fill characteristic and alpha for each pixel 
within the region. For example, a region can be rendered with all pixels having 
the same fill characteristics and varying alphas. In such an embodiment, the fill 
characteristics for the vertices defining the region are the same. These fill 

20 characteristics are then applied to each of the pixels within region. The alpha for 
each of the pixels in region is determined by using interpolation based on the 
alphas for each vertex defining region. 

In one embodiment of the present invention, a computer generates (e.g. 
rendering processor 1 50) and supplies the graphic, by executing instructions from 

25 a program stored in memory. In one embodiment, the computer uses the Open 
GL language and generates a set of polygons using a glBegin, glEnd command 
in conjunction with a GL_QUADS instruction. The GL_QUADS instruction 
provides sets of vertices to the glBegin, glEnd command for drawing 
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quadrilaterals. Also provided are the alphas and fill characteristics for each 
vertex. A quadrilateral is generated by the glBegin, glEnd command for each set 
of four vertices that is provided. 

In an alternate embodiment to the present invention, graphics can be 
5 provided with regions that have more or fewer than four vertices and/or different 
fill characteristics for each of the vertex pixels. When a different number than 
four vertices are employed, the segments that are rendered will be a shape other 
than a quadrilateral. When different fill characteristics are provided for each 
vertex pixel, in one embodiment of the present invention, bilinear interpolation 
10 is employed to determine the fill characteristics for each of the pixels in the region 
based on the fill characteristics for the region's vertex pixels. 

Look back at Figure 12, in step 630, rendering processor 1 50 causes the 
current field to be enhanced. In one embodiment, the enhancement is performed 
by blending the foreground with the background. Example of enhancement 
15 include using a keyer, a video processor or other equipment to combine video 
images or edit program video. 

In further embodiments of the present invention, each pixel in the graphic 
can be analyzed individually to determine its alpha. In one such embodiment, the 
above described process for determining the alpha for vertex can be employed for 
20 each of the pixels in the graphic. 

An alpha value or signal is one example of a blending coefficient A 
blending coefficient is a value used to indicate how to blend one image or video 
with a second image or video. The above discussion describes a means for 
determining alphas for various pixels and using the determined alphas for 
25 blending a graphic using a keyer or a computer. It is contemplated that other 
technologies can be used to blend the graphic and that these other technologies 
may use different blending coefficients than an alpha signal. 

In one embodiment, the lines, shapes or graphics drawn using the 
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telestrator system of the present invention can have varying width, color, texture 
or levels of transparence. Touch screen 104 and display 102 can include a 
graphical user interface which allows the user to select color, line width, texture 
or transparency. Additionally, texture can be a function of zoom. That is, the 
5 more a camera is zoomed in, the more a grass-like texture is added to the 
illustrations. In alternative embodiments, the user interface for determining 
texture, color, line width and transparency is provided in a television production 
studio, on telestration processor 108 or on rendering processor 150. 

Another feature of the present invention includes one touch highlighting. 

10 This allows the operator of touch screen 104 to touch an image on display 102 
and have that image highlighted. The highlight will include a type of graphic to 
identify the object of interest. In one embodiment, the highlight is partially 
transparent so that the object being highlighted is not occluded. In another 
embodiment, the keying process described above can be used to make the 

15 highlight look like it is beneath the object being highlighted. Examples of 
highlights could include a cloud, a hollow circle, icons, team logos, bright spots, 
etc. The user interface could allow the operator to choose the appropriate 
highlight. In operation, the one touch highlight can be performed by the operator 
touching the screen once quickly. Upon sensing the quick touch, the system will 

20 add the pre-chosen highlight to the video at the touched position. In one 
embodiment, the highlight will just be a two dimensional highlight and remain on 
the current video screen only. In another embodiment, the highlight's position 
will be transformed to three dimensional coordinates and appear to be painted 
onto the field. One touch highlighting is performed according to the steps 

25 described above with minor changes. For example, there is no need to smooth or 
thicken a curve. Rather, if the system determines that only one (or a different 
small threshold) point was received, then the system assumes it is adding a one 
touch highlight rather than a line or other shape. A single set of coordinates 
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representing the position of the one touch highlight is transformed in step 3 14 of 
Figure 8. Step 3 16 of Figure 8 is changed to convert the one three dimensional 
location into four vertices that bound the highlight. Alternatively, the system can 
test for a request to highlight by testing for a small amount of time that the finger 
5 touched the screen and/or a maximum length of a curve. 

In one embodiment, the present invention telestration system can be used 
in conjunction with a system for drawing a line on a field of play during an event 
such as described in United States Patent Application No. 09/160,534, filed on 
September 24, 1998. In that combination, delay 158 of Figure 5 could be 

1 0 eliminated or changed. For example, the input to program key er 1 52 would be the 
output from the system drawing the line on the video of the field. That system 
drawing a line would have its own delay. Thus, there may not be a need for 
additional delay or the delay from delay 1 58 can be reduced to a smaller number 
of frames to account for the delay added by the line drawing system. 

1 5 Additionally, in such a combination of systems, many of the components that are 
common to both systems can be shared. 

In one embodiment, the system can perform line smoothing without 
performing the key process or the transformation between coordinate systems. 
In another embodiment, the system can perform the transformation without the 

20 keying and the line smoothing. In yet another embodiment, the system can 
perform the keying without performing the transformations and the smoothing. 
In the embodiments that do not include transformations, the coordinates remain 
in two dimensional space for a particular camera and the graphics drawn on the 
video do not appear to be painted on the field. 

25 The embodiments discussed above contemplate the use of camera sensors 

to determine the attitude of the camera. An alternative to camera sensors includes 
using pattern recognition or other types of image recognition. By looking for 
known locations or known images in a video, telestration processor 108 can 
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determine the attitude of the camera. One example is to use infra red beacons or 
special lights throughout the event. Telestration processor 1 08 can look for these 
beacons or lights in the video. Another alternative is to preselect landmarks in the 
scene prior to the event. The coordinates of these landmarks can be preset. By 
5 searching for these landmarks in the video, telestration processor 108 can 
determine the attitude of the camera. In another embodiment, camera sensors can 
be replaced by pattern recognition allowing the system to recognize landmarks 
near a user's drawings on the video. Then by finding these landmarks in future 
frames or fields, telestration processor 108 can determine the position in the 

10 subsequent frames to place the drawing. Another embodiment of the present 
invention can utilize a combination of camera sensors and pattern recognition to 
place the graphics on the video. One example of such an embodiment uses 
camera sensors to get a rough estimate of where the graphic should be placed and 
uses pattern recognition to refine that estimate. 

1 5 The foregoing detailed description of the invention has been presented for 

purposes of illustration and description. It is not intended to be exhaustive or to 
limit the invention to the precise form disclosed, and obviously many 
modifications and variations are possible in light of the above teaching. The 
described embodiments were chosen in order to best explain the principles of the 

20 invention and its practical application to thereby enable others skilled in the art 
to best utilize the invention in various embodiments and with various 
modifications as are suited to the particular use contemplated. The invention is, 
thus, intended to be used with many different types of live events including 
various sporting events and non-sporting events. It is intended that the scope of 

25 the invention be defined by the claims appended hereto. 
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CLAIMS 

We claim : 

1 1 . An apparatus for annotating video, comprising: 

2 means for receiving an illustration from a user: 

3 means for adding said illustration to said video. 

1 2. An apparatus according to claim 1, wherein: 

2 said video is live video. 

1 3. An apparatus according to claim 1 , wherein: 

2 said video is a replay. 

1 4. An apparatus according to claim 1 , wherein: 

2 said means for adding includes means for transforming said illustration to 

3 a set of three dimensional locations corresponding to an event in real space and 

4 means for transforming said three dimensional locations to a ste of two . 

5 dimensional positions in said video. 

1 5. An apparatus according to claim 1 , further comprising: 

2 means for smoothing said illustration. 

1 6. An apparatus according to claim 1, further comprising: 

2 means for keying said illustration over said video such that said illustration 

3 is drawn on inclusion pixels. 



1 

2 



7. A method for annotating video, comprising the steps of: 
receiving an illustration; and 
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3 adding said illustration to said video. 

1 8. A method according to claim 7, wherein: 

2 said step of adding includes transforming said illustration to a set of three 

3 dimensional locations corresponding to an event in real space and transforming 

4 said three dimensional locations to a ste of two dimensional positions in said 

5 video. 

1 9. A method according to claim 7, further comprising the step of: 

2 smoothing said illustration. 

1 1 0. A method according to claim 7, further comprising the step of: 

2 keying said illustration over said video such that said illustration is drawn 

3 on inclusion pixels. 
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Fig. 1 (prior art) 



2 


0 


1 


5 


r 


s 





T 



10 



Fig. 2 (prior art) 



2 


5 


2 


0 


1 


5 






<— 







WO 01/29681 



PCT/USOO/41280 



2/8 

Fig. 3 
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